Skip to content

LCORE-2860: RHOAI Prow e2e tests fail to start — service pods never become ready#2050

Merged
tisnik merged 1 commit into
lightspeed-core:mainfrom
are-ces:lcore-2860-fix-rhoai-pipeline
Jul 8, 2026
Merged

LCORE-2860: RHOAI Prow e2e tests fail to start — service pods never become ready#2050
tisnik merged 1 commit into
lightspeed-core:mainfrom
are-ces:lcore-2860-fix-rhoai-pipeline

Conversation

@are-ces

@are-ces are-ces commented Jul 2, 2026

Copy link
Copy Markdown
Contributor

Summary

Fix two issues preventing RHOAI Prow e2e pipeline from deploying service pods:

  1. Image build failure (DockerBuildFailed): oc new-build --image=ubi9/ubi-minimal overrode the containerfile's FROM ubi9/python-312 with ubi-minimal, which lacks dnf. Removed the --image flag so the containerfile's own base image is used.

  2. Vector store registration conflict: run.yaml pre-registered the vector store and embedding model under provider_id: faiss, conflicting with the lightspeed-stack enrichment script which registers them under provider_id: byok_e2e-test-docs. Removed the duplicate entries since enrichment handles both.

Type of change

  • Refactor
  • New feature
  • Bug fix
  • CVE fix
  • Optimization
  • Documentation Update
  • Configuration Update
  • Bump-up service version
  • Bump-up dependent library
  • Bump-up library or tool used for development (does not change the final image)
  • CI configuration change
  • Konflux configuration change
  • Unit tests improvement
  • Integration tests improvement
  • End to end tests improvement
  • Benchmarks improvement

Tools used to create PR

  • Assisted-by: Claude Opus 4.6
  • Generated by: N/A

Related Tickets & Documents

Checklist before requesting a review

  • I have performed a self-review of my code.
  • PR has passed all pre-merge test jobs.
  • If it is a core feature, I have added thorough tests.

Testing

  1. Triggered RHOAI Prow periodic pipeline — llama-stack image builds successfully with ubi9/python-312 base
  2. llama-stack pod starts without vector store registration conflict
  3. lightspeed-stack connects to llama-stack successfully
  4. Verified RAG query returns results via /v1/query

Summary by CodeRabbit

  • Chores
    • Simplified end-to-end test and build setup by removing an unused vector store registration and streamlining the image build configuration.
    • No user-facing functionality changes were introduced.

@coderabbitai

coderabbitai Bot commented Jul 2, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

Warning

Review limit reached

@are-ces, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 57 minutes

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro

Run ID: 140ac990-6bc0-4f7a-94cf-e060674aa530

📥 Commits

Reviewing files that changed from the base of the PR and between 1834e01 and cac51f2.

📒 Files selected for processing (6)
  • tests/e2e-prow/rhoai/configs/run.yaml
  • tests/e2e-prow/rhoai/manifests/lightspeed/lightspeed-stack.yaml
  • tests/e2e-prow/rhoai/manifests/lightspeed/llama-stack-prow.yaml
  • tests/e2e-prow/rhoai/pipeline.sh
  • tests/e2e/configuration/server-mode/lightspeed-stack-rhoai.yaml
  • tests/e2e/utils/prow_utils.py

Walkthrough

Modifies the RHOAI e2e test configuration to remove an embedding model and FAISS vector store registration, replacing it with an empty vector store list, and removes an explicit base image argument from the pipeline's build step.

Changes

E2E RHOAI Test Config and Pipeline Updates

Layer / File(s) Summary
Vector store registration removal
tests/e2e-prow/rhoai/configs/run.yaml
Removed the all-mpnet-base-v2 embedding model and its FAISS vector store registration (including embedding dimension/model id and vector_store_id), replaced with an empty vector_stores: [].
Build image argument removal
tests/e2e-prow/rhoai/pipeline.sh
Removed the explicit --image="registry.access.redhat.com/ubi9/ubi-minimal" argument from the oc new-build invocation, leaving the docker strategy to use default image behavior.

Estimated code review effort: 1 (Trivial) | ~5 minutes

Possibly related PRs

Suggested reviewers: tisnik, radofuchs

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Title check ✅ Passed The title clearly identifies the e2e startup failure being fixed and matches the changeset's focus.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
✨ Simplify code
  • Create PR with simplified code

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@tisnik tisnik left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@are-ces are-ces marked this pull request as draft July 2, 2026 11:05
@are-ces are-ces marked this pull request as ready for review July 8, 2026 07:56
- Remove --image flag from oc new-build that overrode the containerfile's
  FROM (ubi9/python-312) with ubi-minimal, which lacks dnf
- Remove duplicate vector_stores registration from run.yaml (enrichment
  handles it)
- Add RHOAI-specific lightspeed-stack config with vLLM defaults
- Parametrize model name via vllm-model-secret so both pods read from
  a single source
- Override inference defaults (default_model/default_provider) in Prow
  config swaps using E2E env vars
- Create /tmp/data before starting lightspeed-stack pod for SQLite

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@are-ces are-ces force-pushed the lcore-2860-fix-rhoai-pipeline branch from 60939c8 to cac51f2 Compare July 8, 2026 07:59
@are-ces

are-ces commented Jul 8, 2026

Copy link
Copy Markdown
Contributor Author

Tests running here

@tisnik tisnik left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@tisnik tisnik merged commit 7d520c9 into lightspeed-core:main Jul 8, 2026
45 of 49 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants